Thermodynamics of genuine non-equilibrium states under feedback control 
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For genuine non-equilibrium states tiiat even at fixed external control parameter exhibit dissipa- 
tion, we extend the Hatano-Sasa equality to processes with feedback control. The resulting bound 
on the maximal extractable work is substantially sharper than what would follow from applying 
the Sagawa-Ueda equality to transitions involving such states. For repeated measurements at short 
enough intervals, the power thus extracted can even exceed the average cost of driving as demon- 
strated explicitly with a simple, analytically solvable example. 
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Introduction and main results.- According to the sec- 
ond law of thermodynamics, the work extracted in 
any process applied to a system coupled to a heat bath of 
temperature T cannot exceed the free energy difference 
(— A_F) between the initial and final equilibrium state. 
If, however, additional information I about the state of 
the system during this process becomes available through 
measurements, more work can be extracted by adjusting 
a control parameter driving the transition accordingly. 
For such processes with feedback, the thermodynamic 
bound 



(1) 



with ks Boltzmann's constant, has been proven for single 
and for repeated measurements both for Hamiltonian and 
stochastic dynamics This relation has been veri- 

fied in an experiment using a colloidal particle trapped 
by two feedback-controlled electric fields Theoreti- 
cal case studies of Eq. ([T]) include Szilard-type engines 
using two-state systems or a partitioning of phase-space 
in two regions 0, 0| and Langevin systems Eli • ^'^^ 
feedback-reversible processes, the bound in Eq. ([T]) can 
be saturated [l^]. For finite-time processes, the maxi- 



mum extractable work has been investigated for a Brown- 
ian particle in time-dependent harmonic traps fisj . Char- 
acteristically, all the systems considered so far reach gen- 
uine equilibrium for a constant external control parame- 
ter A with a Boltzmann-type distribution and zero cur- 
rents. 

On the other hand, there is a large class of systems 
which at constant A approach a genuine non-equilibrium 
steady state (NESS) with permanent dissipation which 
under isothermal conditions is called the housekeeping 
heat Q^^ = TAS^^ [l3l. Examples include essentially 
all motor proteins since they typically move with an on 
average constant speed at fixed external conditions like 
concentrations of chemicals or applied forces or torques 
(see, e.g., [l5l,[3l and refs. therein). Likewise, a colloidal 
particle driven by a constant force along a periodic po- 
tential can serve as another paradigm for such a NESS 
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In this paper, we study the thermodynamics of feed- 
back control of such non-equilibrium states. How much 



(more) work can be extracted if information acquired 
through measurement is used for optimal driving? Is it 
possible to extract net work that exceeds even the cost of 
driving from transitions between such states? One could 
be inclined to try to answer these questions naively by 
generalizing Eq. ([T|) to this situation in the form 



(2) 



with AS**"' being the total entropy change of system and 
heat bath, thereby getting rid off the no longer well- 
defined free energy when dealing with genuine NESSs. 
Indeed; as shown towards the end of the paper, Eq. ([2]) 
is true for such systems even though this fact seems not 
to have been stated yet explicitly. On the other hand, 
Eq. ([2]) is about as useless as a sensitive bound for prac- 
tical situations as the ordinary second law (Eq. ([2]) with 
I = 0) is for non-feedback driven transitions between 
NESSs. The permanent dissipation required to drive 
these NESSs masks all finer details associated with the 
transition between such states. Addressing this issue for 
transitions without feedback, Hatano and Sasa |14| have 
derived an equality from which they obtained a sensitive 
bound focussing on the extra dissipation. 

As our first main result, we will generalize their result 
to feedback driven processes and derive the equality 



(exp[-(As'°* - As'^^ + i)]) = 1 



(3) 



for trajectory-dependent quantities indicated throughout 
by the corresponding small letters and setting = T = 
1 from now on such that heat can be replaced by the cor- 
responding entropy change. The average is over arbitrary 
initial conditions and any number of measurements with 
subsequent adaption of a control parameter implement- 
ing the feedback. This generalization shares some techni- 
cal analogies with Sagawa and Ueda's extension of the 
Jarzynski relation [l9| to feedback control for transitions 
between equilibrium states. Our work, however, applies 
to fundamentally different physical systems, such as ar- 
tificial or biological motors driven by unbalanced chem- 
ical reactions, or basically any other transport problem, 
reaching a NESS at constant control parameter. 
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FIG. 2. Feedback scheme: The energy of the internal state 2 
is changed from iJ > to —E if the system has jumped to 
this state of higher energy, which happens with probabihty 
q{t,n). If state 2 is empty, it can be moved to _E > 0. 



FIG. 1. Model rotary motor: The driving force / > induces 
60°-clockwise rotations along which the internal states 1 and 
2 alternate. The transition rates w'^'^ are given by (O and 



As second main result, for repeated measurements at 
regular intervals, Eq. ([3]) will be shown to imply 



hk 



(4) 



as a bound for the power (which, as any rate, will be de- 
noted by a dot) that can be extracted under isothermal 
conditions from such a feedback driven steady-state ma- 
chine. Here W™ is the cost spent in driving the system. 
For a large enough rate of acquiring information, i.e. for 
I > Q'^**^, a net gain becomes possible since the extracted 
power may then exceed the cost of driving. 

Model system.- We first illustrate this inequality for 
a paradigmatic example for which all relevant quanti- 
ties can be calculated analytically. We consider a system 
which has two internal states labelled 1 and 2 that al- 
ternate in a spatial direction. The NESS is generated 
by a non-vanishing force / > that drives the system 
in a cyclic fashion from state 1 to 2 to 1 to 2 and so 
on. An example of such a system is given in Fig. [TJ 
which is a simplified model for a rotary motor protein 
like the Fl-ATPase IS.Iig. For each clockwise transi- 



tion determined by the rates and W2 , driving the 
system costs the work / (setting for simplicity the dis- 
tance between the states symmetrically to 1), leading to 
a non-zero current. Likewise, the same work is recov- 
ered for any thermally excited step in counter-clockwise 
direction with the rates and w^. Additionally, we 
assume that the states 2 have an energy E higher (or, for 
—E < 0, lower) than the states 1 with energy 0. E will 
serve as the control parameter A. The rates 



and 



Wn 



u;oCxp[(/ + £;)/2] 



(5) 



(6) 



are chosen such that they obey local detailed balance [20[ 
and become maximally symmetric to facilitate the ana- 
lytical solution. Wo being a characteristic frequency of the 
system set to 1 in the following. Maintaining this NESS 
without any feedback interference is easily calculated to 
cost 



W" 



Qh"^ = 2/ 



sinh(//2) 
cosh(£'/2)' 



(7) 



The feedback scheme is implemented as follows, see 
Fig. [2] Perfect measurements at regular intervals sepa- 
rated by a time tm are supposed to reveal whether the 
system is in a state 1 or 2. If the system is in 1, we set the 
energy of state 2 to i? > at no cost since the level is not 
occupied. If at the next measurement the system is still 
in 1 we do nothing. As soon as a subsequent measure- 
ment reveals that the system is in a state 2, we decrease 
the energy of this state to —E < 0, thus extracting the 
work iy°"* = 2E > 0. The next interval then starts 
with the system again in the lower energy state (which 
is now state 2). As soon as a measurement reveals that 
the system has jumped to 1, we raise the energy of the 
now empty state 2 to +E again at no cost. In the steady 
state finally reached in this scheme, the system is as likely 
found in 1 as in 2 on average. The probability ^(t) that 
it will have switched its state at time t after the last 
measurement obeys the master equation 



q{T) = {wt +w^){l~q{T))-{wt 



w^)q{T), (8) 



the solution of which is 

exp{-E/2) 



lir) 



2cosh(£;/2) 



[1 - exp(-7r)] 



(9) 



with the relaxation rate 7 = Wi + + + W2 

The average power extracted from this machine in this 
feedback-driven NESS is 



W°"\t^)^Eq{t^)/tr, 



(10) 
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FIG. 3. Extracted power W^°"' (solid lines) as a function of the 
time tm between two measurements, maximized with respect 
to E (see insert) for the values / = 1 (black) and / = 2 (light 
blue) of the driving force. The corresponding driving power 
VF'" (dash-dotted) and the rate of acquiring information I 
(dashed) are also represented. The two vertical dotted lines 
show the time tm above which the driving requires more power 
than can be extracted, i.e. W°''\im) = W''^{im)- 



since work can be extracted only after a transition from 
1 to 2. Likewise, the rate with which information is ac- 
quired in this binary measurement scheme becomes 

i(i„0 = -[(7lng+(l-g)ln(l-g)]A„ > M^°"'(t„0 (11) 

with q = q{tm). 

Both rates are shown in Fig. [3] as a function of the 
time tm between two measurements and for the energy 
E*{tm, f) which at fixed and / maximizes For 
tml ^ 1 , the system hardly has had a chance to reach the 
higher level. Still, the optimal extracted power W°^^ = 
(4/e) cosh(//2), using i?*(0, /) = 2, is the largest in this 
limit, in which the rate of acquiring information diverges. 
For t„j7 > 1, both W'""* and X tend to and the optimal 
extracted power is obtained for E*{oo, f) ~ 1.278. 

The process with feedback requires less power for driv- 
ing than the one without feedback as given in ([7]). The 
former's power can be determined from the average in- 
stantaneous cost of driving (counting the time r from the 
last measurement) which is 

W"\t) = f[q{r){wt-w^) + {l~q{T)){ujt^w^ )]. (12) 



The time averaged rate 



VV'"(r)dr/t„ 



Shk 



(t,n) (13) 



increases with t,„ as shown in Fig. [3] and is equal to Eq. 
dZl) for tra oo since the system has then reached its 
steady-state. 

In the limit t„il ^ 1, the extracted power exceeds the 
cost of driving for E > /tanh(//2). Such an overall posi- 
tive performance can persist up to a time tm{E, /, 7) but. 



of course, not for arbitrary long intervals between mea- 
surements since the gain per measurement is bounded 
whereas the total cost of driving the system scales lin- 
early with time. 

Proof.- The general results, Eqs. ([3]) and will now 
be proven assuming a Markovian dynamics on a set of 
discrete states n. At time t, the system is in a state n{T), 
jumping at discrete times Tj from state nj~ to state nj~^ . 
A transition between state m and state n occurs with a 
rate Wmnilj) which depends on the instantaneous value 
Ij = A(tj) of a control parameter /(r). The solution of 
the master equation 



drPin, t) 



{l)p{m, t) - ■Wnm{l)p{n, t)] (14) 



for the probability p{n, t) to find the system in state 
n at time r must be distinguished from the stationary 
solution Pn(l) corresponding to the instantaneous value 
of the control parameter. 

Integrated along a trajectory, < r < t, the total 
entropy balance reads [ij, [21 1 



As' 



As'^' + As 



hk 



As" 



(15) 



Here, As^^^ = — lnp(??t,t) + lnp(no,0) is the change in 
system entropy with n(0) = uq and n{t) = rit- The 
entropy change associated with the heat bath is the sum 
of housekeeping entropy As^^ and excess entropy 

As-= fdrJ25ir-r,)\n[p:^^Alj)/K,-m (16) 
Jo ,j 

where the sum runs over all jumps in the corresponding 
time interval of length t. 

For processes without feedback, this excess entropy 
obeys an integral fluctuation theorem 



g("t) 
Po{no) 



exp(-As'=^) ) = 1 



(17) 



valid for any initial condition po{n) and an y n ormalized 
function g{n) evaluated at the end point jl3 . |2]| . By 
choosing for po and g the stationary distribution cor- 
responding to the initial and final value of the control 
parameter I, respectively, this relation becomes precisely 
the Hatano-Sasa relation [l^. For the following proof, 
we will, however, keep Po('^) '"^ a-n arbitrary initial con- 
dition. We choose g{n) = 5n.m which leads, by using 
Bayes' theorem, to the conditioned average 

(po\no)exp(-As'='')|nt =TO))p(m,t) = 1 (18) 

valid for any m from which we get the equivalent 

^(exp(-As<='^)|no,nt =m)p(m,i) = 1 (19) 

no 

by conditioning the average also on the initial state and 
thus cancelling the term p^^(ri,o). 
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For the process with feedback, we first consider the 
case of two measurements at time ti and t2 with out- 
comes 2/1 and j/2, respectively. The trajectory-depend_ent 
information acquired with the first measurement is 



11 = ln[pi(ni,ti|?/i)/p(ni,ti)] 



(20) 



where pi{ni,ti\yi) is the probability to find the system 
in state ni = n{ti) given the measured value yi. This 
function now serves as the initial condition for the fur- 
ther time evolution of pi{n, T\yi) which obeys the master 
equation ((T4| with a protocol /(r) that depends on j/i. 
The second measurement yields the information 

12 = In[p2(n.2,t2|y2,2/i)/pi(n.2,i2|j/i)] (21) 

with the analogous definition of P2('^2j ^2 1?/2, 2/i) and 
n{t2) = 7T.2- In this process with feedback, the total 
change in system entropy becomes 



As 



\np2{nt,t\y2,yi) +\npo{no). (22) 



The total change of excess entropy can be split into three 
contributions from the three intervals i = {0 < r < 
ti},ii ~ {ti <T< 12} and in ~ {t2 < t < t} as 



As" 



As'r 



(23) 



With 1 = ii -I- 12 and by combining Eqs. ([TS1) and (|20t - 
[23|) . the average on the left hand side of Eq. (|3]) still 
conditioned on the result y2 , yi can be written as 



1 ^-Asr -P(^i.^i) g-A^;? 



Poino) 



Piini,ti\yi) 



X e 



Pi{n2,t2\yi) 

P2{n2,t2\y2,yi) 

'P2{'nt,t\y2,yi) 



^ \po ?^o 

mi ,7712 V 



m = mi ) p(mi,ii) 



X (e \ni = mi, 712 = m2) Pi{m2,t2\yi) 



x(e '^^■^■P2(?^t,i|y2,yi)|n2 = TO2)... = 1 (24) 



where we have introduced conditioned averages on the 
three intervals i, ii and Hi (indicated by subscripts) 
which eliminate the explicit factors Pi^{ni,ti\yi) and 
P2^{n2,t2\y2,yi)- The term denoted by a is 1 for any 
mi due to Eq. (fTS]) . Likewise, the subsequent summa- 
tion of (3 over mi is 1 due to Eq. ^T9\\ as is the remaining 
sum of 7 over m2. It should be obvious that each fur- 
ther measurements will lead to another summation of 
;9-type and hence Eq. ([3]) is proven for any number N 
of measurements. This proof shows that Eq. ^ holds 
true even as an average conditioned on a fixed record 



{Ua, 1 ^ a < N}. In practice, a further average over all 
possible results {ya} will typically be used as we also will 
for the remaining discussion. 

Applying Jensen's inequality to Eq. ^ leads to 



AS"'"* < -Q 



hk 



(25) 



for the ensemble averages. For isothermal processes, we 
now invoke the first law 

^in _ ^out ^ ^ Qhk ^ gcx ^ ^ ^gtot _ ^gsys ^ 

(26) 

The splitting of the total work W into a driving one 
and an extracted one depends on the specific system and 
is not fixed by these thermodynamic considerations. If 
N ^ 1 measurements are repeated at times separated 
by t„i, inserting (|26l) in (|25|) and then dividing by Ntm 
leads to the bound (|4]) since both AE and A^"'^'' are fi- 
nite boundary terms which vanish in the limit of a steady 
state operation. 

As an aside, we note that the same reasoning applies 
if we start with the valid modification of Eq. (|17p where 



is replaced by As'^^ + As [21[. As a result, one 



obtains a concise proof of the Sagawa-Ueda equality Q 
(exp[— (As'°* -I- 1)]) = 1 which is thus shown to hold true 
even for feedback-driven transitions between the genuine 
non-equilibrium states considered here. The resulting 
bound ([2]), is, of course, much weaker than ([25l) since 
the housekeeping heat is extensive in time. 

Concluding perspective - The equality ([3]) and the re- 
sulting inequalities (j4]) and (|25p generalize and sharpen 
previous results for feedback-driven systems with time- 
dependent Hamiltonians or potentials to transitions be- 
tween genuine non-equilibrium states. Searching for the 
optimal protocol following a measurement is an issue that 
so far has primarily been asked for the former systems. 
It should now be investigated within this much wider 
class encompassing natural and artificial molecular ma- 
chines driven by (or delivering output from) biochemical 
reactions. Our example of a simple driven two-state "in- 
formation machine" has shown that its power becomes 
maximal if the time between measurements is the small- 
est and that it can even exceed the cost of driving. We 
suspect that both statements have a broad range of valid- 
ity but for exploring its limits more analytical and numer- 
ical work will be necessary. Finally, the biggest challenge 
might be to come up with a first experimental demon- 
stration of these relations. Whether the paradigmatic 
colloidal particle driven along a periodic potential or a 
molecular motor is more suitable for such an experiment 
remains to be seen. 
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